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Abstract. This study aimed to investigate 
LDE mathematics and science questions in 
terms of cognitive requirements (knowing, 

applying and reasoning) and structural 
properties (conceptual, algorithmic and 
graphical). The methodology adopted in 
the current study was document analysis. 

The results of this study indicated that LDE 

science assessments emphasized conceptu- 
al questions, while mathematics questions 
were more algorithmic in structure. In addi- 
tion, both mathematics and science items 
de-emphasized graphical representations. 
In terms of their cognitive requirements, 
both science and mathematics LDE items 
neglected the cognitive domain of reason- 
ing. Moreover, science questions mostly 
met in the intersection of knowing and 
conceptual, while mathematics questions 
often required students to apply knowledge 
on algorithmic questions. This study also 
concluded that the distribution of question 
requirements differed from year to year. 
Some implications regarding the results 
included that implementing a nation-wide 
assessment that neglects graphical repre- 
sentations contradicts the target goal of the 
mathematics and science programs in Tur- 
key. The fact that the LDE was designed to 
determine the level of students’ learning yet 
neglects reasoning questions also presents 
a contradiction to the curricular aims. 
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Introduction 


In today’s world, governments allocate significant portions 
of their budget to education, raising expectations for educational 
efficiency, which is typically measured through variables such as 
classroom materials, quality of teaching, quality of educational 
environment, and students’ beliefs. However, a great deal of em- 
phasis has also been placed on determining students’ achieve- 
ment levels and comparing them from year to year (Berberoglu 
& Kalender, 2005). Responsible for preparing students for life, 
schools must teach and improve abilities like problem solving, 
critical thinking, and reading skills. A higher quality social life is 
promoted in an educational system that produces academically 
qualified individuals. While determining and evaluating achieve- 
ment, countries usually assess their students’ performance through 
standardized national and international examinations. Beside shar- 
ing the progress of their students and comparing their results at 
the international level, most countries, including Turkey, use the 
examinations for students transitioning to higher educational 
institutions, like middle school to high school or high school to 
college. The limited quota of highly regarded high schools causes 
intense rivalry, and systems have been established to identify the 
most excellent students. 

The level determination examination (LDE) in Turkey is taken 
by eighth grade graduates seeking entrance to secondary educa- 
tion institutions. New teaching and learning programs at primary 
education institutions in Turkey became effective in 2004, and 
in the 2007-2008 academic year, a new system was applied to 
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secondary education (MONE, 2007a). The main purpose of the new system was to diminish the burden 
of the entrance examination and emphasize the importance of school itself. The old transition system 
relied heavily on a placement exam known as Secondary School Placement Exam (SSPE). This test caused 
anxiety for students and parents: evaluating three years of work through a 120-minute test was not in 
line with evaluation principles of Turkey’s new teaching and learning program, and SSPE only covered 
some parts of the curriculum, prompting students to neglect uncovered subjects (MONE, 2007b). 

Many comparative studies have investigated examination systems of different countries. Recent 
studies investigated such factors as achievement in examinations (Incikabi, 2012; Kogkar & Gen¢oz, 2004), 
gender related issues (Garner & Engelhard, 1999; Lisle, Smith, & Jules, 2005), reasons for failure (Kim & 
Dembo, 2000; Kjellstr6m & Pettersson, 2005; Lukacs & Tompa, 2002), and alignment between curriculum 
and examinations (Azar, 2005; Incikabi, 201 1a, b; Liang & Yuan, 2008; Ozmen, 2005; Saderholm &Tretter, 
2008). The results of these studies address compatibility between content of examinations, curriculum 
coverage, and classroom instruction, since exam content shapes curriculum by affecting instruction. 
According to Kim (2005), differences in distribution of the contents in the assessment may cause poor 
performance, since students tend to disregard topics not emphasized by the examinations. Similarly, 
Kasanen and Raty (2008) have highlighted how national assessments affect student attitudes towards 
lessons by causing them to neglect project and performance activities and practice more on question 
types in the tests. Therefore, test specifications gain importance in evaluating education efficiency. 

Test specification plays an important role in interpreting international comparisons based on test 
scores (Linn, 2003). Over or under-emphasis of certain cognitive processes or topics may be of advan- 
tage to some countries (Ben-Simon & Cohen, 2004). Item format also affects achievement (Ben-Simon 
& Cohen, 2004). Although early versions of international competencies included multiple choice items, 
current competencies employ a variety of formats (multiple choice, structured response, essay) due 
to a global demand for authenticity in testing and the desire to assess more complex, higher-order 
cognitive processes. 

Recent studies of question types (Basturk, 2011; Costu, 2007, 2010; Erkan Erkoc, 2011; Kim & Pak, 
2002; Maloney, 1994) have indicated that the intensity of graphical, conceptual, and algorithmic questions 
in examinations has changed classroom instruction by affecting both students as learners and teachers 
as instructors. The studies comparing students’ performances on graphical, conceptual and algorithmic 
questions are very few in number compared to those focusing on the performance on conceptual and 
algorithmic questions (Costu, 2007, 2010). Erkan Erkog (2011) compared pre-service teachers’ scores 
on conceptual, algorithmic, and graphical questions and indicated achievement on conceptual ques- 
tions. Moreover, Costu (2007) presented that eleventh-grade students performed better on conceptual 
chemistry questions than algorithmic and graphical chemistry questions. In another study Costu (2010) 
conducted with twelfth-grade students, algorithmic questions were the only question types with high 
scores. However, researchers need to initiate studies of national assessments of developing countries 
such as Turkey in terms of process requirements applied to international competencies and question 
structures. 

Based on above literature, this study aimed to investigate LDE mathematics and science questions 
in terms of their cognitive requirements (knowing, applying and reasoning) and structural properties 
(conceptual, algorithmic and graphical). Being in line with the aim, the following research questions 
were to be sought for answers within the scope of the current study: 

e How were various cognitive domains, structural properties, and their intersection distributed 
in the mathematics questions of LDE? 

e How were various cognitive domains, structural properties, and their intersection distributed 
in the science questions of LDE? 


Methodology of Research 


In this study, document analysis was applied to LDE mathematics and science questions. Document 
analysis is known as an effective method for systematic review (Cohen, Manion, & Morrison, 2007). 
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Sample of Research 


The target was eighth grade LDE questions. A total of 160 questions from the years from 2009 
to 2012 (80 for mathematics and 80 for science) were assessed. During the academic year 2007-2008, 
when the new mathematics program became effective, LDE was applied first to sixth and seventh grad- 
ers. Thereafter, the secondary institutions selection and placement examination (SSPE) was replaced by 
the LDE examination for eighth graders. 


Procedures 


To interpret the LDE science and mathematics questions, four experts (two in the field of 
mathematics education and two in science examination) convened in two groups. All experts had 
familiarity and experience with the assessment and its framework. The expert panel members had an 
opportunity during the opening session to review, classify, and discuss several practice items in order 
to establish acommon understanding of classification procedures. The researchers did not develop the 
coding themes: they were adapted from the literature (Costu, 2007, 2010; Erkan Erkoc, 2011; Nakhleh, 
1993; Nakhleh & Mitchell, 1993) and Trends in International Mathematics and Science Study (TIMSS) 2011 
mathematics and science frameworks (Mullis, Martin, Ruddock, O’Sullivan, & Preuschoff, 2009). Table 1 
presents the coding themes and related behaviors/explanations. During coding, the panel members 
coded each question independently. Coders classified items in the most detailed way possible—ideally, 
to the objective level, that in cases where items appear to address multiple themes (See Appendix A for 
sample LDE items and their codings). The initial coder agreement rate was 85 percent for mathematics 
and 87 percent for science. Each item for which the coders did not agree then was discussed until an 
agreement was reached on how the item would be coded. 


Table 1. Coding categories used in the study. 


Categories Sub-categories Behaviours 
Conceptual Use conceptual information regarding the given condition in the question 
Structural Properties Algorithmic Use algorithmic calculations to reach a numerical value 
Graphical Interpret/retrieve information from a graphic 
Knowing Recall, Recognize, Compute, Retrieve, Measure, Classify/Order 
Cognitive Require- Applying Select, Represent, Model, Implement, Solve Routine Problems 
ments (Mathematics) 
Reasoning Analyze, Generalize, Synthesize/Integrate, Justify, Solve Non-routine Problems 
, Recall, Recognize, Define, Describe, Illustrate with Examples, Demonstrate 
Knowing 


Knowledge of Scientific Instruments 


Cognitive Require- 
ments (Science) 


Compare, Contrast, Classify, Use Models, Relate, Interpret Information, Find 


nppwying Solutions, Explain 


Analyze, Synthesize, Integrate, Hypothesize, Predict, Design, Draw Conclusions, 


poaecng Generalize, Evaluate, Justify 


Data Analysis 


The frequency of cognitive domains and structures of the questions, broken down by examina- 
tion year, were determined. The interpretation of the analyses is descriptive in nature. 
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Results of Research 
The results of the current study were introduced with regards to the research problems. 
LDE Science Questions 


Table 2 shows the distribution of LDE science questions based on TIMMS cognitive domains and 
structural properties. The majority (about 89%) were structured conceptually, while 8 percent were 
graphical and only a few (about 3%) required algorithmic procedures. Among the cognitive requirements 
used, knowing and applying were mostly highlighted, with a slight emphasis on knowing. However, 
only one-tenth of the science questions required reasoning, which is the highest order of and the most 
complex cognitive domain of all. 


Table 2. Distribution of LDE science questions based on TIMSS science cognitive domains and 
structural properties. 


f % 

Conceptual 71 88.75 

Structural Properties Graphical i 6.15 
Algorithmic 2 Zo 

Knowing 38 47.5 

Cognitive Requirements Applying 34 42.5 
Reasoning 8 10 


Table 3 shows the distribution of dual qualifications (cognitive and structural) across the science 
questions in LDE. Almost half were included in the intersection of knowing and conceptual codes, 
while about one-third placed in the conceptual-applying combination. In addition, the questions in 
the intersection of conceptual-reasoning, probably the most selective questions in the examinations, 
covered only one-tenth of the all science items. Moreover, all graphical and algorithmic questions were 
limited to the cognitive requirement of applying. 


Table 3. Distribution of cognitive domains and structural properties in LED science questions. 


Cognitive Requirements 


Applying Knowing Reasoning 
Conceptual 25 (31.25%) 38 (47.5%) 8 (10.0%) 
Structural fs . 
Properties Algorithmic 2 (2.5%) 0 0 
Graphical T (8.75%) 0 0 


Figure 1 shows the percentage distribution of LDE science items across the years in terms of 
structural properties and cognitive domains. Among structural properties, only the 2010 LDE included 
algorithmic questions. Moreover, conceptual questions were the most represented (more than 75% 
per year). Among the cognitive domains, applying and knowing were the most emphasized. Although 
reasoning questions were included for the last three years, they were few in number compared to other 
domains. 
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Percentage distribution of question types and cognitive domains from 2009 to 2012. 


LDE science questions’ coverage of learning areas as stated in the science curriculum in Turkey 
(MONE, 2005b) is shown in Table 4. Physical events was the most covered learning area (about 33%) 
in the science part of the examinations. Except for one physical-event question coded as algorithmic- 
applying, all other physical-event questions were structured as conceptual questions focused on know- 


ing and applying. 
Table 4. 


Learning Areas 


Physical Events 


Matter and Change 


Life and Living Beings 


The Earth and the Universe 


Qualifications 


Algorithmic 
Graphical 
Conceptual 
Algorithmic 
Graphical 
Conceptual 
Algorithmic 
Graphical 
Conceptual 
Algorithmic 
Graphical 


Conceptual 


Knowing 


Distribution of LDE items in terms of science content domains. 


Applying 


1 


11 


Total (%) 


26 (32.5) 


24 (30) 


24 (30) 


6 (7.5) 


The learning areas Matter and Change and Life and Living Beings covered 30 percent each of all 
science questions. For Matter and Change, LDE questions highly emphasized conceptual questions; there 
were few graphical and algorithmic questions. Conceptual Matter and Change questions required know- 
ing (mostly), applying (some), and reasoning (only one), whereas graphical and algorithmic questions 
only called for applying. Life and Living Beings included only conceptual (mostly) and graphical (few) 
questions. Although the conceptual questions had almost equal distribution among knowing, apply- 
ing, and reasoning, the graphical questions solely incorporated applying. The Earth and the Universe 
was least covered (about 8%) and included only conceptual questions categorized under knowing and 


applying cognitive domains. 
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LDE Mathematics Questions 


The distribution of LDE mathematics questions in terms of TIMSS cognitive domains and struc- 
tural properties is shown in Table 5. Three quarters of mathematics questions were algorithmic, while 23 
percent were conceptual, and only a few (about 3%) graphical. Among the cognitive requirements used 
during the solution process, more than half of the items (about 56%) required applying, whereas about 
one-third were categorized as knowing. As with the LDE science questions, few (about 13%) called on 
reasoning, the highest order of and most complex cognitive domain. 


Table 5. Distribution of LDE mathematics questions based on TIMSS science cognitive domains 
and structural properties. 


f % 
Algorithmic 60 15 
Types of Questions Conceptual 18 22.0 
Graphical 2 25 
applying BB 
Cognitive Domain Knowing 25 Olea 
Reasoning 10 125 


Table 6 shows the percentage distribution of qualifications (cognitive and structural) across the 
mathematics questions. Almost half of the LDE mathematics items (about 54%) were categorized in the 
intersection of algorithmic and applying, while a few placed in algorithmic-knowing and conceptual- 
knowing, each at 15 percent. In addition, questions in the intersection of conceptual-reasoning, prob- 
ably the most selective questions, covered only about 6 percent of all mathematics items. Moreover, 
no mathematics item with graphical structure required reasoning. 


Table 6. Distribution of cognitive domains and structural properties in LDE mathematics ques- 
tions. 


Cognitive Requirements 


Applying Knowing Reasoning 
Algorithmic 43 (53.75%) 12 (15%) 5 (6.25%) 
aa Conceptual 1 (1.25%) 12 (15%) 5 (6.25%) 
Graphical 1 (1.25) 1 (1.25%) 0 


The percentage distribution of LDE mathematics items across the years in terms of structural 
properties and cognitive domains was presented in Figure 2. Among the structural properties across 
all years, algorithmic questions were most represented (more than 60% per year). Although conceptual 
questions were included in each LDE, they were few in number compared to algorithmic questions. On 
the other hand, few algorithmic questions were included in LDE examinations in 2009 and 2010. Among 
the cognitive domains, applying was the most emphasized across the years, followed by knowing. Simi- 
lar to the science section, LDE mathematics sections also de-emphasized the reasoning questions that 
require a higher order of thinking skills, such as generalizing, justifying, and analyzing. 
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Figure 2. Percentage distribution of question types and cognitive domains from 2009 to 2012. 


Table 7 shows the percentage distribution of LDE mathematics questions based on the learning 
areas as defined by the curriculum in Turkey (MONE, 2005a). Measurement was the most covered learn- 
ing area (about 33 percent). Except for two questions coded as conceptual-knowing and conceptual- 
reasoning, all others were algorithmic questions focused on applying (mostly), reasoning (a few), and 
knowing (a few) cognitive requirements. 


Table 7. Distribution of LDE items in terms of mathematics content domains. 


Learning Areas Qualifications Knowing Applying Reasoning Total (%) 
Algorithmic 1 21 2 

Measurement Graphical - - - 26 (32.5) 
Conceptual 1 - i 
Algorithmic 1 1 

Geometry Graphical - - - 14 (17.5) 
Conceptual 7 1 4 
Algorithmic 5 6 - 

Numbers Graphical - - - 14 (17.5) 
Conceptual 3 - - 
Algorithmic - 12 2 

Algebra Graphical - - - 14 (17.5) 
Conceptual - - - 
Algorithmic 5 4 - 

Probability and Statistics Graphical 1 1 - 12 (15) 

Conceptual 1 - - 


Geometry, Numbers, and Algebra covered 18 percent of all mathematics questions. Geometry was 
the only learning area that emphasized conceptual questions. There were a few algorithmic geometry 
questions, while no geometry question included use of graphical representations. Conceptual geometry 
questions required knowing (mostly), reasoning (some), and applying (only one), whereas algorithmic 
questions required knowing and reasoning. Numbers included mostly algorithmic questions and some 
conceptual questions. Although the algorithmic questions had almost equal distribution across knowing 
and applying, the conceptual questions solely called for knowing. Algebra included only algorithmic 
questions that required applying (mostly) and reasoning (few). Probability and Statistics was the least 
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covered (15 percent) and included mostly algorithmic questions with a few graphical and conceptual 
questions. The algorithmic and graphical questions were categorized in knowing and applying, while 
the conceptual question required knowing. 


Discussion 


This study aimed to provide an analysis of the mathematics and science items included in LDE as- 
sessments in Turkey based on (a) cognitive requirements (knowing, applying and reasoning) as defined 
in TIMSS 2011 and (b) structural properties (algorithmic, conceptual and graphical). The study was 
limited to 160 questions from the LDE (80 in mathematics, 80 in science). 

An overview of results is provided in Table 8. In terms of distribution of structural properties, LDE 
science assessments emphasized conceptual questions, while mathematics questions were more al- 
gorithmic in structure. Both mathematics and science items de-emphasized graphical representations. 
Van Dyke and White (2004) have stated that graphical interpretation supports abstract thinking skills. 
Standard examinations such as LDE need to be designed to involve reading, forming, and interpreting 
graphics (Forster, 2004). A number of studies indicate that students have common deficiencies in these 
areas (Ates & Stevens, 2003; Beichner, 1994; Berg & Smith, 1994; Kekule, 2008; McDermott, Rosenquist, 
& van Zee, 1987; Saglam-Arslan, 2009). For example, in his two-stage study that was first performed 
with 480, then 700 students, Kekule (2008) discovered that students describe graphics as an outline 
or picture of reality. Berg and Smith (1994) also reported that students perceive graphics as a picture 
instead of a symbolic depiction of knowledge. 


Table 8. Overview of the results. 


Science Questions Mathematics Questions 
Emphasized De-emphasized Emphasized De-emphasized 
Structural Properties Conceptual aes Algorithmic Graphical 
P F Algorithmic g P 
Cognitive Requirements anos Reasoning Applying Reasoning 


Applying 


Conceptual-knowing Algorithmic- applying 


Dual Qualifications ; The others The others 
Conceptual-applying 
Conceptual Graphical Algorithmic Conceptual 
Distribution Across Time Knowing Algorithmic Applying Graphical 
Applying Reasoning Knowing Reasoning 


Both science and mathematics LDE items neglected the cognitive domain of reasoning, which 
requires that students analyze, generalize, synthesize/integrate, justify, or solve non-routine problems 
(Mullis et al., 2009). Science questions mostly met in the intersection of knowing and conceptual, while 
mathematics questions often required students to apply knowledge on algorithmic questions. These 
differences in cognitive balances may affect student performance in other competencies (such as 
TIMSS) that include more reasoning questions (Ben-Simon & Cohen, 2004). For instance, Cil and Cepni 
(2012) observed low performance of Turkish students on questions requiring correlational cognitive 
and hypothetical thinking abilities. 

This study found that the distribution of question requirements differed from year to year. Some 
representation types (such as conceptual in science and algorithmic in mathematics) existed in each as- 
sessment, while some (algorithmic in science and graphical in mathematics) were habitually neglected. 
A similar misbalance was also evident in the distribution of cognitive requirements by each assessment. 
For example, questions requiring reasoning were not included in 2009 LDE science items and were de- 
emphasized over the years. 
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Conclusions 


A general overview of the research findings might lead a conclusion that LDE does not assess 
students’ knowledge in terms of the aspects that were investigated in this study. Taking into account 
of the complexity of learning process, it is a compulsory need for assessments to interrogate students’ 
learning from different angles. Since, assessments that highlight only one cognitive procedure or con- 
sist of one item type would affect classroom instruction, shape the curriculum (Kim, 2005) and cause 
students’ neglecting the other cognitive processes. 

Upon consideration of the results of the study, some implications should be taken into account by 
policy makers and test/curriculum designers. Following the reform movement, the adjusted curriculum 
in Turkey emphasizes using multiple representations in teaching (MONE, 2005a, 2005b); however, imple- 
menting a nation-wide assessment that neglects graphical representations contradicts the target goal. 
Moreover, the science and mathematics programs in Turkey put great emphasis on improving students’ 
problem solving and critical, creative, and reflective thinking in line with the behaviors of the reasoning 
domain as used in this study. The fact that the LDE was designed to determine the level of students’ 
learning yet neglects reasoning questions also presents a contradiction to curricular aims. 

The results of this study are beneficial to researchers who investigate national assessments, 
as well policymakers and curriculum designers who interpret the results of such exams. Further studies 
investigating the test-curriculum-teaching triangle will reinforce the findings of the current study. 
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Appendix A 
Sample LDE Items 


Table 9. Sample codes for selected LDE science items 





Qualification 
Items ue 
Structural Cognitive 
Properties Requirements 

Some concepts regarding Natural processes and their definitions are as follows: Conceptual Knowing 

¢ Aftershock 

¢ Foreshock 

; Concepts 

¢ Intensity 

¢ Magnitude 

¢ The degree of damage on buildings and people 

caused by the earthquake. 
¢ Small earthquakes occurring before the main shock. Definitions 
¢ The value of the ground motion measured by seismo- 
graph. 
Which concept does remain outside when the concepts are paired with their definitions? 
A) Aftershock B)Foreshock C) Intensity D)Magnitude 
(LDE 2012, p. 15, Q7) 
The table shows the running hours (during a month) of a vacuum cleaner having 2000 watt 
power and an iron having 2200 watt power: 
The running times (hour) 
vacuum cleaner iron 

1st week 4 2 

2nd week 5 

3rd week 2 : Algorithmic Applying 

4th week - 3 
According to table, what was the total electric energy (kwh) consumed by the vacuum 
cleaner and the iron during this month? 
A)21 B)22 C)42 D)44 
(LDE 2010, p. 15, Q8) 

Quantity of heat (J) 

a quantity of substance (g) 
Graphical Applying 


The graph shows the change in the evaporation point with regard to the quantity of a pure 
substance that is at the point of boiling temperature. What is the numerical value of “?” 
given in the graph? 


A) (5204/10):20 B) (5204/10+20) 
C) (5204/20) 10 D) 5204-(10+20) 


(LDE 2010, p. 15, Q14) 
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Gulay, conducting the experiment given below, records her observations and notes the 





results. 
vinegar _| 
baking 
powder 
My experiment: 
| have added vinegar on baking powder. Conceptual Reasoning 
My observations: 
Balloon was bulged, cap got warmed, and liquid was obtained in the cap. 
My results: 
In this experiment, what result can Gulay reach? 
A) Vinegar and baking powder preserved their chemical structure. 
B) The amount of baking powder did not change. 
C) The vinegar did not react with the baking powder. 
D) All liquid that was obtained was vinegar. 
(LDE 2010, p. 15, Q10) 
Table 2. Sample codes for selected LDE mathematics items. 
Qualification 
Items - 
Structural Cognitive 
Properties Requirements 
Table: Basketball players’ average points per game and average point range 
Name of the player Average point per game Point range 
Cemil 17 3 
Alper 17 15 
Hasan 12 15 
Ali 12 Q 
Graphical Knowing 
Table shows the average points and point range of the players who played the same 
number of game last year. Which player did score more points with the least changing 
amount? 
Ali B) Hasan C)Alper D) Cemil 
(LDE 2009, p. 12, Q16) 
A bakery sold a total number of 144 patty and pastry during the day and made 144TL 
profit from the sell. How many pastries have been sold if the price for a patty anda 
pastry is 50 Kr and 75 Kr, respectively? 
Algorithmic Applying 


80 B)64 C)58  D)44 


(LDE 2009, p. 13, Q19) 
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Algorithmic Reasoning 





Taking out which cube do not cause change in the surface area of the solid given in the 
figure that consists of unit cubes. 


(LDE 2011, p. 11, Q13) 


A person who investigates which color car has been mostly sold found out that white 
cars were preferred the most. What measure was utilized to reach this conclusion? 


Median B)Mode C) Mean D) Range Conceptual Knowing 


(LDE 2011, p. 11, Q16) 
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